Narrative Visualizations Best Practices and Evaluation: A Systematic Literature Review
¶Abstract: In recent years there has been a growing interest in integrating data visualizations into narrative stories, as an effective way of conveying information and knowledge. By leveraging the best practices established in the literature, narrative visualizations can reduce the cognitive workload associated with chart comprehension. However, it is also a critical and challenging to assess the results; several methodologies have been proposed to assess visualizations. In this article, we present a systematic literature review of ninety-five data storytelling and information visualization studies. Our goal is to collect and summarize current definitions of “data storytelling” reported in the literature, the best practices for designing narrative visualizations as well as evaluation criteria and methods to assess them. We contribute by deriving a working definition of data storytelling, distinguishing among the concepts involved and provide an overview of design guidelines to assist practitioners and researchers to create narrative visualizations. In addition, we characterize the main evaluation criteria and methods. Our findings highlight the need for more out-of-the-box, ready to use evaluation tools that allow a rapid and iterative assessment of narrative visualizations.
Keywords: information visualization, data storytelling, narrative visualizations, evaluation, systematic review.
Data visualization has become essential to understand large datasets and communicate findings. As a subfield of visualization research [1], Information Visualization focuses on the visual representations of abstract data [2][3] to enhance understanding and to support and amplify cognition [4].
Storytelling has been used for a long time as an effective way of conveying information and knowledge. Stories aid memory and recall by embedding information into characters, settings, relationships, and events [5], and a narrative is what gives shape to a story [6]; hence the importance of integrating data visualization into narrative stories. In structured contexts, researchers can use these stories to support discussion, decision making and process analysis [7].
Narrative visualizations that leverage best practices established in the literature can reduce the cognitive workload associated with chart comprehension [8], [9] and prompt positive decision-making [10]. However, the development and dissemination of guidelines for their design has been scarce. As demand for narrative visualizations increases, so does the need for standards to support their creation. By understanding the impact of specific visual encodings on performance, we can assist end users in making informed, effective decisions.
In recent years, evaluation has emerged as a central and challenging issue in the field of visualization [11], [12]. There is a diverse set of qualitative and quantitative methods for evaluating different aspects of data-driven stories [11], some of which include controlled experiments, usability tests and case studies [13]. But these methods only focus on a visualization’s ability to communicate facts. Recent studies by Dimara et. al. suggest that people can make irrational decisions even when they properly understood the data [14] and good performance with analytic tasks does not guarantee the same in decision making[15]. Researchers aim to move beyond this evaluation approach to assess the utility of a visualization. As Matzen et. al. [16] indicate, it would be valuable to have evaluation tools that can be deployed rapidly and iteratively during the design process to assess visualizations prior to conducting a user study.
Motivated by this, we present a systematic literature review (SLR) of ninety-five data storytelling and information visualization studies. Our goal is to collect and summarize current definitions of “data storytelling” reported in the literature, best practices for designing narrative visualizations as well as evaluation criteria and methods to assess them.
The rest of this paper is organized as follows: Section 2 summarizes background and related works, Section 3 describes the methodology for conducting this SLR, Section 4 reports the results, Section 5 presents the discussion for each research question as well as the threats to validity, and Section 6 concludes and outlines future work.
This section presents an overview of prior research relevant to our work, including existing SLRs, and contextualizes the aims of this study.
Over the last two decades there has been a growing interest in visual storytelling and data visualization to communicate findings. There are studies that review the use of decision support systems in fields such as agriculture [17], where visualization plays a key role in assisting end-users to interpret the data, or environmental sciences [18], where it is necessary to interact with actors outside of the scientific community and therefore presents the challenge of effective communication to enable users to generate actionable understanding.
In the field of Information Visualization, various authors have attempted to define storytelling. Segel and Heer [19] introduced the concept of “narrative visualizations” and highlighted that data stories differ from traditional storytelling – not every visualization can be thought of as a visual data story [20]. In [21] Hullman and Diakopoulos build upon this definition and describe the techniques involved in the communication of data stories.
Riche et. al. [22] use the term “data-driven story” for stories that are either based on or contain data, visualized to support one or more intended messages, usually including annotations (labels, pointers, text) or narration [20]. The main idea that all these definitions share is the use of visualizations along with narratives to enhance communication and promote insight.
Several researchers have discussed emerging opportunities and challenges in storytelling in the field of visualization. Kosara and Mackinlay [23] give an overview of the topic, highlighting the importance of storytelling for data analysis and presentation, as visualizations are increasingly used for decision making. In [20], the authors take a narrower view of what data storytelling involves, to facilitate discussion around it, and present a comprehensive view of the storytelling process. Ojo and Heravi [24] examined 44 cases of award-winning data stories to identify storytelling practices and characterize the tools and techniques to create them.
More recently, El Outa [25] proposed a conceptual model of data narrative for exploratory analysis to support the entire process of building a data story. They also provide structured definitions of the key concepts in the domain of data narrative. Finally, in [26], Matei and Hunter review existing definitions of “story” and “storytelling” and present examples of what constitutes a “good” story. They outline a methodology for constructing effective, data-based stories.
Many experts in chart design provide guiding principles to create clear and efficient visualizations with the lowest possible cognitive load [10], [27]–[29]. Based on their studies on graphical perception, Cleveland and McGill [30], [31] suggest a ranking of graphical devices for displaying quantitative data. In [32], Kosslyn describes eight psychological principles for the design of charts that respect the cognitive, perceptual and memory processes of the human brain.
Tufte, a well-known advocate of minimalistic design, argues that visualization is about simplifying complex information and defines “graphical excellence” as displays that communicate with clarity, precision, and efficiency [28]. He proposes the data-ink ratio and suggests that all ink that is not used to present data should be removed.
Several empirical studies have evaluated the effectiveness of different guidelines, such as: limiting the use of gridlines[33], [34], using color strategically [35]–[39], adding labels and annotations[40], [41], avoiding visual clutter [42], [43], or allowing the viewer to cluster information and recognize patterns[44], [45]. Some authors have investigated the effect of chart embellishments [47] – [50] and emphasis techniques[50], [51], while others focused on a specific type of chart: bars[53] – [57], scatterplots [58] – [61] and treemaps [61], [62].
Other research report on the emerging challenges and opportunities with respect to visual analytics. In [63] the authors reviewed the entries for the Visual Analytics Science and Technology (VAST) Challenge, as well as guidelines developed by other researchers and synthesized the results into an initial set for assessing environments and analytics reports. From a product design perspective, Adagha et. al. [64] point out that part of the challenge is that current visual analytics tools do not have standardized design criteria and processes, thus raising the question of what the attributes of effective tools are and how to measure their impact.
Despite such efforts, to the best of our knowledge no study has systematically gathered the best practices involved in the design of narrative visualizations.
Evaluation is attracting considerable interest in the field of information visualization and some studies characterize the reported approaches through literature reviews. Isenberg et. al. [65] reviewed 581 papers and provided a quantitative and objective report of the types of evaluation practices encountered in the visualization community. They found an increasing trend in the evaluation of user experience and user performance but though it has improved over the years, the general level of rigor of reporting evaluations is still low.
Lam et al [66] provide an overview of the different types of evaluation scenarios, categorized into those for understanding data analysis processes and those which evaluate visualizations themselves. They base their categorization on questions and goals, rather than existing methods, encouraging the community to consider the context before choosing an evaluation method.
In [67] the authors summarize types and components of visualizations used in the health industry and methodologies for development and evaluation of patient-facing visualizations. They suggest that there is a need for greater attention to developing these kinds of visualizations, since the evaluation methods had little to no standardization across the articles, thus making it difficult to identify and compare best practices. They identified three key opportunities: more robust data collection and reporting; consistent evaluation approaches, and interpretation enhancement.
Tory and Möller [68] discuss the use of formal laboratory user studies against alternative evaluation techniques in HCI, such as focus groups, field studies and expert reviews, particularly, heuristics evaluations. They argue that such reviews are valuable ways to assess visualizations.
Borgo et. al. address specific methodologies for evaluating visualizations. In [69] they reviewed the use of crowdsourcing experiments for empirical evaluation of visualizations reported in 82 papers. They summarize the design, methods, tasks, tools, and measures used in this kind of studies and found that many papers failed to properly report relevant aspects about the experiment. They present a taxonomy of practices along with a checklist to support researchers in reporting tasks.
As for evaluation criteria, Bertini, Tatu and Keim [70] presented a systematic analysis of quality metrics to support the exploration of high-dimensional data sets and defined a quality metrics pipeline. Shah and Hoeffner [71] review graph comprehension and provide a set of guidelines to design charts that enhance interpretation and discuss unresolved questions related to data presentation and graphical literacy.
Saket et. al. [72] review visualization evaluation in terms of user experience and characterize the goals and metrics relevant to storytelling and narrative purposes such as memorability, engagement, and enjoyment. More recently, in [73] the authors argue that data stories must address different challenges depending on the context. They provide a non-exhaustive set of criteria and evaluation methods by which data-driven stories can be assessed.
Based on the review outlined above, the goal of this work is to define “data storytelling” in the context of Information Visualization and to collect and analyze guidelines for the creation of narrative visualizations. This can help to develop a better understanding of the design process and serve as a starting point to derive improvement recommendations.
We also provide an overview of the evaluation criteria to get a clearer view of what is considered an “effective” visualization for narrative purposes, as well as the reported evaluation strategies. Our work is based on Lam et. al. visualization scenario [66], as we are interested in characterizing evaluation on a “final product” perspective. We do not consider conventional methods such as user studies and experiments, but rather focus on finding methodologies that are consistent with the best practices and criteria found.
In this section we describe the steps of the SLR process, following Kitchenham and Charters guidelines [74]. These include the definition of a review protocol to ensure rigor and reproducibility, in which we determine: research questions, data sources and search strategy, inclusion and exclusion criteria, quality assessment, data extraction and selected studies. Additionally, we followed another two guidelines for conducting automated searches [75] and effective data extraction [76]. All study materials, including supplementary files can be accessed at [LINK].
As stated in previous sections, the goal of this SLR is to identify, analyze and summarize existing definitions of “data storytelling,” best practices for the design of narrative visualizations as well as evaluation criteria and methods. For this purpose, we formulated the following research questions:
RQ1: Is there a valid and accepted definition of “data storytelling”? This question seeks to establish what exactly data storytelling is and whether it has a formal and accepted definition for both academics and practitioners.
RQ2: What are the data storytelling best practices reported in the literature and how are they implemented? The goal of this question is to summarize the guidelines reported in the literature to create effective narrative visualizations and to find practical demonstrations of how they are applied.
RQ3: What are the criteria to evaluate narrative visualizations? The various stakeholders involved in the data storytelling process may have different goals according to their perspectives [77]. This research question aims at identifying the criteria by which those goals are met that can be used to evaluate narrative visualizations.
RQ4: What are the current strategies to evaluate narrative visualizations? This question identifies the methods by which the criteria defined in the previous question can be evaluated and their characterization: what types of charts they apply to, the metrics they use, the tools to support them, etc.
This study covers four main types of visualizations, namely: line charts, bar charts, scatter plots, and pie charts, in addition to choropleth maps, area charts, bubble charts and treemaps. We focus on these visualizations as they are the most frequently occurring types, according to [78] and [79]. Throughout the study, we use the term “chart” instead of “graph” to avoid confusion with the field of graph drawing.
The search and selection process of the primary studies was performed in three steps to control the number and relevance of the results: automatic search, study selection, and snowballing search.
Database search: We conducted database searches on three indexing systems related to software engineering: ACM Digital Library, IEEE Xplore and Scopus. We chose these sources as they are considered standard libraries [80]. The search string was divided into two parts: one containing a keyword that describes our main subject and its synonyms and the other focusing on the different topics of the research questions.
Table 1: Syntax of the search string for each digital library.
| Search Engine | Search String |
|---|---|
| Scopus | TITLE-ABS-KEY (( "data storytelling" OR "data-driven storytelling" OR "data visualization" OR "data story" OR "data-driven story" OR "information visualization" OR "data-driven visualization" ) AND ( practice OR guideline OR guide OR principle OR evaluation OR assessment OR metric OR measurement OR criteria OR goal OR characteristic ) |
| IEEE | ("All Metadata":"data story" OR "All Metadata": "data visualization" OR "All Metadata": "information visualization") AND ("All Metadata":practice OR "All Metadata":guide* OR "All Metadata":principle OR "All Metadata":heuristic OR "All Metadata":evaluation OR "All Metadata": assessment OR "All Metadata": metric OR "All Metadata":measurement OR "All Metadata": criteria OR "All Metadata": goal OR "All Metadata":characteristic) NOT ("All Metadata":optical) |
| ACM | Abstract:("data storytelling" OR "data-driven storytelling" OR "data visualization" OR "data story" OR "data-driven story" OR "information visualization") AND Title:(practice OR guideline OR guide OR principle OR heuristic OR evaluation OR assessment OR metric OR measurement OR criteria OR goal OR characteristic) |
The search was limited to title, abstract and keywords. The specific implementation of the search string for each database is presented in Table 1 while Table 2 presents each term together with its keywords.
Table 2: Main terms and synonyms used to create the search string.
| Main Term | Keywords |
|---|---|
| Data storytelling | "data storytelling” OR “data-driven storytelling" OR “data story” OR "data-driven story" OR "data visualization" OR "information visualization” |
| Best practice | “best practice” OR practice OR guideline OR principle |
| Criteria | criteria OR goals |
| Evaluation | evaluation OR assessment |
Snowballing search: We complemented the database search with forward and backward snowballing. The goal of this step was to expand the set of relevant papers by focusing on papers citing or being cited by previously included studies [81]. Articles collected during this stage were also added to the main list and selected according to the inclusion/exclusion criteria.
We define the following inclusion (I) and exclusion (E) criteria based on the SLR guidelines proposed by [74] to select the appropriate studies and filter out the unrelated ones. We were interested in primary studies published in any year up until 2021 that present some contribution on data storytelling, visualization best practices and evaluation.
- I1: The title, abstract and keywords explicitly state that the paper is related to data storytelling and data visualization.
- I2: The study is a full paper with empirical evidence.
- I3: The paper is peer-reviewed (journal article, conference paper)
- I4: The full text of the paper is available.
- E1: The paper is not written in English.
- E2: The paper’s full text is not accessible.
- E3: The paper is a gray publication without peer review.
- E4: The paper is explicitly a short paper.
- E5: The paper does not cover any of the visualizations described in Section 3.2.
We found a total of 11.818 articles ranging from the years 1984 to 2021 by applying the search strategy defined in Section 3.3. The search was conducted using title, abstract and indexed keywords (see Table 3).
Table 3: Automated search details.
| Database | Search results |
|---|---|
| ACM Digital Library | 1.635 |
| IEEE Xplore | 243 |
| Scopus | 9940 |
| Total | 11818 |
The procedure for studies selection consisted in five phases, presented in Fig. 1. In Phase 1, we obtained the studies from electronic databases, downloaded the results, and organized them into a single spreadsheet. Out of the 11.818 search results, 8476 were unique. In Phase 2, we checked the title of each primary study using the I/E criteria. Articles unrelated to our subject were omitted and 5321 titles were kept for consideration in the next phase. In Phase 3, we checked the abstract of each primary study and excluded 3845 papers. If there was insufficient data, the paper was left for the next phase. In Phase 4 the remaining papers were analyzed using the full text. In this step, we obtained 87 relevant papers and excluded 1389. The selected articles were reviewed for quality assessment in Phase 5 before final inclusion in this SLR. Another 13 papers were included via snowballing, resulting in 95 studies overall.
Fig. 1: Study selection flowchart.
An identity code was assigned to every individual study. The list of papers with their identity code is available in Table A 1 (see Appendix).
In addition to the inclusion/exclusion criteria, it is critical to assess the quality of primary studies[74]. The Cochrane Reviewers’ Handbook [82] suggest that quality relates to the extent to which the study minimizes bias and maximizes internal and external validity. The quality assessment (QA) of selected studies was achieved by a scoring technique to evaluate their credibility, completeness, and relevance. All papers were evaluated against a set of 11 quality criteria. The assessment instrument is presented in Table 3. Questions Q1, Q2, Q4-Q11 were adopted from the literature, while question Q3 is proposed.
Each quality assessment question is judged against three possible answers: “Yes” (score = 1), “Partially” (score = 0.5) or “No” (score = 0). The quality score for a particular study is computed by taking the sum of the scores of the answers.
Table 4: Quality assessment checklist.
| ID | Question |
|---|---|
| Q1 | Is there a clear statement of the goals of the research? |
| Q2 | Is there sufficient discussion of related work? |
| Q3 | Are the visualizations under study clearly described? |
| Q4 | Is the purpose of the analysis clear? |
| Q5 | Is the investigation process adequately documented? |
| Q6 | Are the statistical methods described? |
| Q7 | Are the study participants or observational units adequately described? For example, SE experience, type (student, practitioner, consultant), nationality, task experience and other relevant variables. |
| Q8 | Are all study questions answered? |
| Q9 | Is there a discussion about the results of the study? |
| Q10 | Are the limitations of this study explicitly discussed? |
| Q11 | Are the lessons learned interesting and relevant for practitioners? |
To extract data from the selected primary studies we used the template shown in Table 5. Collected data includes general information (title, authors, year of publication and source) and information related to the research questions. Before the actual data extraction, we performed an extraction pilot with a random set of ten papers to calibrate the instrument, reinforce the extraction strategies, and avoid misunderstandings.
We decided to extract the information exactly as the authors mentioned it. For each paper, we considered abstract, introduction, methodology, results, and conclusion. In some cases, a comprehensive reading of the paper was necessary. Any conflicts were discussed and resolved internally by the authors to reduce bias and ease reproducibility. To measure the level of agreement between researchers we used the Cohen Kappa statistic [83].
Table 5: Data extraction form.
| Focus | Item | Description |
|---|---|---|
| General Information | Identifier | Reference number given to the article |
| Bibliography | Author, year, title | |
| Source | Journal/Conference | |
| Aim | Goal of the study | |
| Type of study | Empirical strategy | |
| RQ1 | Data storytelling definition | Definition of data storytelling or related terms |
| RQ2 | Best practice | Recommended practice or guideline |
| Application | Ways to implement the guidelines and best practices | |
| RQ3 | Criteria | Evaluation criteria for data visualizations |
| Assessment | Techniques to assess if the evaluation criteria were met | |
| RQ4 | Evaluation method | Strategy to evaluate data visualizations |
| Type of chart | The visualization technique covered by the evaluation method | |
| Metrics | Values measured by the evaluation method | |
| Tools | Software applications, models and algorithms used to support evaluation |
To support this task, we used Atlas.ti [84] a software for conducting qualitative research that allows to highlight, annotate, and code data segments of interest. For RQ2 we used an open and axial coding strategy based on grounded theory [85]. First, we read each guideline and assigned it a best practice (BP) id, such that a new BP was created for guidelines that did not resemble previous ones. Then, we used axial coding to compare the best practices to each other and identify categories or themes, relying on general knowledge and categorizations proposed by other authors. Similarly, for RQ3, we organized the criteria in two levels (main and sub criteria) grouping similar criterions into one.
The quality assessment helped us increase the reliability and achieve a coherent synthesis of results [86]. We present the results of the QA in Table A 2 (see Appendix) according to the questions described in Table 4. The results show that the overall quality of the studies is high since the mean quality was 90%.
The selected primary studies were published between 1984 and 2021. Fig. 2 presents the number of studies by year of publication. Overall, we found at least one study in each year since 2005. An increasing number of publications is observed since 2010, with the majority of them conducted between 2013 and 2021. We found the highest number of studies in 2018. This evidences a trend in the topic of information visualization and data storytelling.
Fig. 2: Distribution of selected primary studies over the years 1984 – 2021.
The selected studies were classified according to the six categories defined by Wieringa et. al. [87]: i) validation research, ii) evaluation research, iii) solution proposal, iv) philosophical papers, v) opinion papers, and vi) experience papers. Table 6 shows the classification by type of research.
Table 6: Research type categorization [87].
| Type | Description | # of papers |
|---|---|---|
| Validation research | Techniques investigated are novel and have not yet been implemented in practice, e.g.: experiments, work done in the lab. | 77 |
| Solution proposal | A solution for a problem is proposed. The solution can be either novel or a significant extension of an existing methodology. The potential benefits and the applicability of the solution is shown by a small example or a good line of argumentation. | 15 |
| Philosophical papers | These papers sketch a new way of looking at existing things by structuring the field in form of a taxonomy or conceptual framework. | 3 |
| Evaluation research | A technique is implemented in practice and an evaluation of it is conducted. Implementation of the technique is shown in practice and its consequences are demonstrated. | 0 |
| Experience papers | Experience papers explain what and how something has been done in practice. It must be the personal experience of the author. | 0 |
| Opinion papers | These papers express the opinion of somebody whether a certain technique is good or bad, or how things should have been done. They do not rely on related work and research methodology. | 0 |
Validation research (81%) and solution proposal (16%) were the most adopted research type within the selected papers. Few philosophical papers appeared over the years and none of the selected studies belonged to the evaluation, opinion, and experience papers categories.
Over the last years, there has been a growing effort by the visualization community to better understand and characterize data storytelling. The broader use of the term suggests that complementing interactive visualizations with narratives can make the data exploration a more engaging and memorable experience [19], [21], [23], [88], [89].
In 2010, Segel and Heer [S02] introduced the term Narrative Visualizations, to describe “an emerging class of visualizations that attempts to combine narratives with interactive graphics”. They draw a distinction between data stories and traditional storytelling: stories are often presented in a linear, controlled progression of events, while data visualization can be organized according to different structures and can also be interactive, promoting new questions and explorations. They analyzed 58 examples of narrative visualizations and identified the techniques for telling stories with data, such as common genres (magazine style, annotated chart, partitioned poster, flow chart, comic strip, slide show, and video) and design patterns (the martini glass structure, interactive slideshows, and drill-down stories). According to the authors, narrative visualizations put data at the forefront of storytelling, resulting in visualizations with a storytelling component intended to convey a specific message.
In S03, Hullman and Diakopoulos further discuss this concept by exploring the rhetorical devices used in narrative visualizations. They suggest that “narrative visualizations typically rely on a combination of persuasive, rhetorical techniques to convey an intended story to users as well as exploratory, dialectic strategies aimed at providing the user with control over the insights gained from interaction”. In their study, the authors examined 51 examples of narrative visualizations to understand how the design and rhetorical choices affect user’s interpretation of a story and highlight the nature of narrative visualizations as “multimedia artifacts that can’t be reduced to visualization alone.”
In 2013, Hullman et. al. [S47] define the storytelling focus in visualization research as “the study of narrative visualizations, development of automated data storytelling tools and proliferation of narrative visualizations in news media attest”. They argue that conveying a narrative with visualizations requires context definition, information selection and choosing an order in which to present visualizations; and provide a focused analysis of 42 data stories to understand the forms that structure and sequence take. In line with [S02], they state that in a narrative visualization, “the events of interest are patterns in data sets represented in visualizations”, and the author of the story must decide how to organize the representations into a compelling and understandable sequence.
Dimara et. al. suggest in [S60] that the narrative component in visualizations can provide benefits in terms of enhanced motivation, attention and engagement, and this could translate into improved task comprehension and higher-quality responses.
In S08 the authors analyze existing open data platforms and suggest the need to leverage storytelling as means to support the discovery of relevant data and the generation of insights, thus improving transparency and trust. They consider data storytelling as “the process of translating data analysis into simple, logical stories that can be understood by non-technical audiences.”
More recently, Liem et. al [S80] empirically tested storytelling techniques to make visualizations compelling, engaging and persuasive. They explored whether certain practices influence people’s attitudes towards anthropomorphized data graphics (the practice of visualizing data about people in a way that helps the audience relate [90]), and though the results were inconclusive, it shows the potential of storytelling in the field of visualization research.
We found a total of 38 best practices that are summarized in Table 7. Across the ninety-five studies there were a few authors that proposed categorizations regarding different aspects of visualizations, such as [21], [91]–[93]. Based on that and following the strategy described in Section 3.7, we classified the best practices into five categories, as follows:
Cognitive: Cognition is defined as the way our brain processes information. Visual cognition involves deriving meaning from external representations of visual information[94]. This category encompasses the guidelines and recommendations that aid to reduce the cognitive load associated with chart comprehension.
Data: This set of best practices involves aspects related to the data sources behind the visual representation and it is concerned with the information itself: transparency, reliability, and accuracy.
Perceptual: Cleveland and McGill define “graphical perception” as the visual decoding of the quantitative and qualitative information encoded on graphs [30]. It is directly related to the user’s ability to interpret information. Perception-based guidelines allow us to display data in a way that the important patterns stand out [95]. This category includes practices related to the human visual system, such as the mapping of visual variables, the strategic use of color, among others.
Presentation: This category incorporates choices about how the data will be mapped to the visual domain and thus it is also related to perception. It considers the aesthetics and style of the representation, the layout of the elements, as well as the clarity and effectiveness of the design.
Usability: Usability is described as the degree to which a product can be used (ISO 9241). This category is concerned with the functionality and understandability of charts, including features such as accessibility, consistency and interactivity that allows the users to perform tasks effectively.
A given practice may be part of more than one group, since some aspects such as cognition and perception are intrinsically linked. The “Context” column indicates the scope of the practice in terms of the charts it applies to. The “Reference” column specifies the primary study in which the guideline was found (see Table X in Supplemental Materials for visual references).
Table 7: Best practices found in the literature.
| ID | Category | Best practice | Context | Reference |
|---|---|---|---|---|
| BP1 | Cognitive | Simplify complex ideas | General | S03, S08, S16, S91 |
| BP2 | Cognitive | Provide contextual information | General | S23 |
| BP3 | Cognitive | Use metonymy to map visual signs to implicit meanings. | General | S03, S23 |
| BP4 | Cognitive | Limit the number of series or categories displayed | General; Line chart | S37, S42 |
| BP5 | Cognitive Data | Select the appropriate visualization considering the types of data to represent and the advantages and disadvantages of each technique. | General | S12, S15, S21, S46, S65, S84, S91, S94 |
| BP6 | Cognitive Data | Present uncertainty information alongside point estimates to enable informed judgments and decisions. | General | S24, S54 |
| BP7 | Cognitive Data | Use a common baseline to make comparison between series easier | Line chart Bar chart | S37, S50 |
| BP8 | Cognitive Data Perceptual | Map information and data dimensions to the most salient visual features | General | S03, S12, S23, S30, S32, S33, S40, S42, S51, S84 |
| BP9 | Cognitive Data Perceptual | Avoid obscuring information and confusing. | General | S02, S03 |
| BP10 | Cognitive Perceptual | Grids and labels should be usefully visible; effective in providing information without being obtrusive. | General | S13, S27, S39, S50, S72 |
| BP11 | Cognitive Perceptual | Provide redundancy to improve comprehension and memorability of the information. | General | S16, S57, S69 |
| BP12 | Cognitive Perceptual Presentation | Focus the important data points of a visualization to draw the user's attention to the relevant areas. | General | S02, S03, S30, S47, S76, S78, S81, S90, S95 |
| BP13 | Cognitive Perceptual Presentation | Declutter visualizations by removing unnecessary elements such as grid lines, marks, legends, and colors. | General | S10, S13, S19, S39, S81 |
| BP14 | Cognitive Perceptual Presentation | Maximize the data-ink ratio by limiting the amount of "ink" that is not used to present data. | General | S13, S34, S82, S83 |
| BP15 | Cognitive Presentation | Use text, labels and annotations for effective information consumption and decision making. | General | S02, S19, S23, S25, S49, S50, S57, S60, S92 |
| BP16 | Cognitive Presentation Usability | Avoid adding embellishments, non-essential imagery, and chart junk to not distract readers from data. | General | S34, S43, S56 |
| BP17 | Cognitive Usability | Choose the visualization technique that better supports the expected tasks. | General | S01, S20, S21, S24, S35, S40, S42, S45, S51, S67, S69, S74, S75, S77, S87, S93, S94 |
| BP18 | Cognitive Usability | Guide the user in the consumption of the story. | General | S02, S06, S47, S62 |
| BP19 | Cognitive Usability | Tailor designs to the needs of their audience, taking into account individual user characteristics. | General | S41, S42, S52, S85 |
| BP20 | Data | Provide credits of the provenance of the data and details of the design to ensure transparency and credibility | General | S03, S08, S23, S80, S91 |
| BP21 | Data | Avoid omitting information | General | S03, S24 |
| BP22 | Perceptual | Use visual imagery and embellishments to help to fix a chart in a viewer’s memory | General | S23, S34, S43, S56, S57, S81 |
| BP23 | Perceptual | Use color effectively, preserving important differences in the data | General | S22, S30, S64, S74, S79, S90 |
| BP24 | Perceptual | Group similar marks and visual features to facilitate tasks | General | S42, S55 |
| BP25 | Perceptual | Incorporate tangible or situated feelings to evoke senses and create experiences | General | S23 |
| BP26 | Perceptual Presentation | Avoid using 3D effects | General | S03, S29, S34, S50, S83, S88 |
| BP27 | Perceptual Presentation | Manage design parameters with care | General | S12, S17, S39, S67, S72 |
| BP28 | Perceptual Presentation Usability | Use comfortable font sizes to improve legibility | General | S23 |
| BP29 | Perceptual Usability | Make information accessible to impaired users | General | S23, S78, S95 |
| BP30 | Presentation | Communicate a narrative in a clear way | General | S02, S08, S23, S62 |
| BP31 | Presentation | Layout the elements of the charts and the whole story so that it's easier to understand the data. | General | S25, S26, S62, S75 |
| BP32 | Presentation | Define the sequence of events and possible story paths | General | S47, S62 |
| BP33 | Presentation | Maintain consistency throughout the story | General | S02, S47 |
| BP34 | Presentation | Define what role the visualization plays in the story | General | S62 |
| BP35 | Usability | Include interaction techniques to allow the user to explore the data. | General | S02, S03, S04, S08, S29, S62, S68 |
| BP36 | Usability | Provide feedback to show to readers that their input affects the story and to improve the performance of specific encodings. | General | S62, S68 |
| BP37 | Usability | Allow the user control over the story components. | General | S02, S62 |
| BP38 | Usability | Avoid lengthy tutorials and explicit description of every design manipulation | General | S03, S06 |
Fig. 3 presents the best practices grouped according to the categories defined above. It provides a clear visual of the guidelines that fall under more than one category.
Fig. 3: Sankey diagram of the best practices grouped into categories.
Regarding the implementation of best practices, for the vast majority we found several ways to apply them. For BP4, BP26 and BP38 we did not find any practical demonstrations. Overall, there were 384 implementations, varying from two to more than fifty per best practice (see Fig. 4).
Fig. 4: Number of implementations found per best practice.
Taking the first three best practices for instance, Table 8 shows their implementations. The “Context” column indicates the scope of the implementation (e.g.: “bar charts”, “line charts” or “general” if it applies to all types of charts) and the “Reference” column specifies the primary study where it was found. In some cases, one implementation could serve as practical application of more than one practice. For the complete set of guidelines and implementations, see Table X (Supplemental Materials)
Table 8: An example of best practices implementations.
| ID | Description | Implementation | Context | Reference |
|---|---|---|---|---|
| BP1 | Simplify complex ideas | Keep distracting or irrelevant information to a minimum | General | S03 |
| Select the variables to visualize to create a subset of a larger data set to present a simplified visual representation of chosen features | General | S03 | ||
| Averaging techniques like mean, median, and clustering substitute simpler representations for a wider range of values. | General | S03 | ||
| Use textual and visual summaries | General | S03 | ||
| Categorize, bin or aggregate | General | S03 | ||
| Allow users to aggregate, combine or merge datasets that are similar or complementary | General | S08 | ||
| Use appropriate visual embellishments to present complex information and help viewers grasp key concepts more effectively. | General | S16 | ||
| BP2 | Provide contextual information | Messaging provides essential instructions, guidance, or contexts for the data visualizations. | General | S23 |
| Provide a detailed explanation for data visualization | General | S23 | ||
| BP3 | Use metonymy to map visual signs to implicit meanings. | Use suggestive spatial mappings like “left = past, right = future” or “up = more or better, down = less or bad” | General | S03 |
| Pair visualized patterns to categories via typographic and color mappings | General | S03 | ||
| Use bright/dark color for positive/negative tone | General | S23 | ||
| Use warm/cold color for positive/negative tone | General | S23 | ||
| Use high/low colorfulness to express positive/negative tone | General | S23 | ||
| Use semantically-resonate colors | General | S23 |
We identified 47 evaluation criteria and organized them in two levels: 5 main criteria and 42 sub criteria. Table 9 presents the definition of the main criteria along with the sub criteria it comprises.
Table 9: Evaluation criteria and sub criteria.
| ID | Description | Definition | Reference | Sub criteria |
|---|---|---|---|---|
| C1 | Comprehension | A visualizations’ ability to ease understanding of the underlying information and generate insight[93]. | S11, S27, S32 | SC01; SC02; SC03; SC04; SC05; SC06; SC07; SC08; SC09; SC10; SC11; SC12; SC13; SC14; SC15; SC16. |
| C2 | Engagement | A combination of factors that allow to keep the user’s attention and interest to view or interact with the visualization [96][73]. | S08, S54, S62, S63 | SC01; SC17; SC18; SC19; SC20; SC21; SC22; SC23; SC24; SC25; SC26; SC27; SC28. |
| C3 | Information | Relates to the quality and credibility of underlying data [97]. | S59 | SC29; SC30; SC31; SC32; SC33; SC34. |
| C4 | Memorability | It deals with the retention of information and relevant aspects of the visualization. A memorable visualization “sticks” in the viewers mind [48]. | S16, S35, S49, S57, S58 | SC01; SC35; SC03; SC25; SC36. |
| C5 | Usability | The degree to which the visualization helps users accomplish goals with minor effort; it comprises the functionality and understandability of charts [92]. | S24, S59 | SC01; SC17; SC35; SC37; SC38; SC39; SC40; SC41; SC42. |
Sub criteria is described in Table 10. In both cases, the ID column identifies each criterion (we used C for criteria and SC for sub criteria); the Definition column explains the concept as extracted from the primary studies and the “Reference” column specifies the primary study in which the criterion was found (see Table X in Supplemental Materials for visual references).
Table 10: Definition of sub criteria for evaluation.
| ID | Description | Definition | Reference |
|---|---|---|---|
| SC01 | Aesthetics | Visual appeal (attractiveness) of a chart. | S05, S29, S52, S53, S61, S62, S63, S11 |
| SC02 | Complexity | The amount of detail in the visual representation. | S10, S16, S28 |
| SC03 | Uniqueness | The quality that makes a chart visually distinct from others. | S23, S57, S11 |
| SC04 | Identity | The degree of differentiation between data series in a shared space or data points in a visualization. | S37 |
| SC05 | Affordance | Aspects of visualizations that suggest how it should be used; a visual clue to the functions and usage of an object [98]. | S11 |
| SC06 | Clarity | Degree of coherence of a chart. | S11 |
| SC07 | Cognitive load | The mental effort of processing information [10]. | S50 |
| SC08 | Confusion | The degree of difficulty the users experience when processing information. | S19 |
| SC09 | Dynamism | Motion or animation methods. | S11 |
| SC10 | Legibility | Readability of the design “at a higher scale”. | S11 |
| SC11 | Mapping | If the properties of the visual representation most closely match the information being represented [98]. | S11 |
| SC12 | Perspective | If the visualization allows multiple views (perspectives) of the same information. | S11 |
| SC13 | Reachability | Navigation, exploration mechanism of a visualization system. | S11, S61 |
| SC14 | Saliency | Indicates how a target stands out from the background. | S28 |
| SC15 | Simplicity | The quality of designs that display only relevant and necessary information. | S11 |
| SC16 | Space management | Describes whether space is “shared” or “split” between time series. | S37 |
| SC17 | Ease of use | Perceived usability of the design. | S05, S53, S62 |
| SC18 | Focused attention | The concentration of mental activity [99], [100] | S61, S62 |
| SC19 | Novelty | Features of the interface that users find unexpected, surprising, new, and unfamiliar [100], [101]. | S61, S63 |
| SC20 | Autotelism | The chart is an end in and on itself. | S61 |
| SC21 | Challenge | The amount of effort experienced by the participant in performing a task[100]. | S61 |
| SC22 | Control | How “in charge” users feel over their experience with the technology [100]. | S61 |
| SC23 | Creativity | The degree to which a visualization promotes the generation of new thoughts or ideas. | S61 |
| SC24 | Endurability | The assessment of users’ perception of success with a task, and their willingness to use an application in future or recommend it to others [100]. | S62 |
| SC25 | Expressiveness | Denotes the design traits that make infographics vivid, eloquent, and story like. | S23 |
| SC26 | Felt involvement | The feeling of users of being drawn into and involved in a task and the overall assessment of the experience as “fun” [100]. | S62 |
| SC27 | Interest | The “feeling that accompanies or causes special attention to an object or class of objects” [100]. | S61 |
| SC28 | Visual flow | The congruence between the way a reader navigates the story, the visual components of the story, and the type of visual feedback the reader receives; along with the nature of the data and facts that the author wants to communicate. | S62 |
| SC29 | Insight | Nontrivial discovery about the data. Addresses the need for users to understand information. | S08, S61, S73 |
| SC30 | Concreteness | Indicates the degree of pictorial resemblance of a visual representation. | S16 |
| SC31 | Confidence | The degree to which a visualization helps a user trust in his/her understanding of the data set. | S73 |
| SC32 | Essence | How a visualization communicates the essence of the data set with respect to overview and context. | S73 |
| SC33 | Usefulness | Value, relevance of the visualization. | S53 |
| SC34 | Visual Intelligence Density | The amount of useful information that a decision maker obtains by interacting with a visualization. | S14 |
| SC35 | Familiarity | The frequency with which some visualization and visualization features are encountered. | S05, S06, S16 |
| SC36 | Significance | Indicates the relationship between what is depicted in the visual representation and the function it refers. | S16 |
| SC37 | Aspect ratio | The ratio between the width and height of a diagram. | S17 |
| SC38 | Communication | Relates to the quality of human-computer interaction. It comprises aspects of user’s satisfaction, flexibility, and learning. | S58 |
| SC39 | Efficiency | Perceived productivity (e.g.: time and effort saved) | S53 |
| SC40 | Intersections | Intersections caused by a guiding line. | S19 |
| SC41 | Time | Captures how a visualization facilitates faster, more efficient understanding of data with respect to both searching and browsing. | S73 |
| SC42 | Visual clutter | Unnecessary elements and embellishments that diminish a visualizations’ ability to communicate. | S37, S19 |
We extracted the criterions exactly as the authors mentioned in the studies and then grouped those that were similar or expressed related ideas into a single one, such that “appearance” = “aesthetic”, “discovery” = “insight”, attention = “focused attention”, and so forth.
Fig. 5 displays the number of sub criterions per category, considering that certain criterions overlap across the categories.
Fig. 5: Number of sub criterions per category.
Several methodological approaches have been developed for evaluating visualizations. The most common are empirical studies where participants perform benchmark tasks and researchers collect measures like completion time, error rate and accuracy. But as [102] and [103] point out, these methods only focus on data “facts” and fail to consider other aspects such as decision-making support, or insight generation. The motivation of this RQ was to move beyond formal laboratory studies and focus on evaluation models that allow for a more holistic assessment.
The work by Zhu et. al. [S28] presents a complexity analysis method that can be used during the design stage prior to conducting user studies. They measure the complexity of a visualization in terms of visual integration, number of separable dimensions for each visual unit, the complexity of interpreting the visual attributes, and the efficiency of visual search. These factors indicate the amount of cognitive load for a given design.
Padda et. al. [S31] propose to evaluate a visualization based on its comprehension support, i.e., if the users are able to grasp the underlying intentions of the design. They derive a set of criteria and characterize them into three aspects, namely: perception, cognition, and visual interface. In S67, the authors present a methodology to compare two specific techniques (scatter plot and line chart) and determine which one is better suited for a given problem based on a “consistency score”. They measure consistency between a LOESS fit to the data and the visual representation by the two visualization types. The visualization with the larger consistency score is then suggested to be the right choice.
Another approach for evaluating and guiding the design of visualizations is the use of heuristics, which is a common usability inspection method [104]. Researchers have tested and developed various sets of heuristics that capture issues specifically related to visualization and decision making. Forsell and Johansson [S36] empirically compared six sets of heuristics and synthesized them into a new set of ten heuristics that offer a wide explanatory coverage of common issues in information visualization. Similarly, Dowding and Merrill [S70] developed a heuristic checklist that combines Nielsen’s usability principles along with visualization-specific heuristics, (including those of Forsell and Johansson’s), to evaluate data visualization dashboards.
S71 presents an instrument that integrates a set of heuristics from the literature to evaluate visualizations, that also serves as a best practice checklist for their design. Wall et. al. [S73] develop a methodology based on the value framework proposed by Stasko [105] to evaluate visualizations along four components: time, insights, essence, and confidence. They derived a set of value-driven heuristics, with the goal of estimating and quantifying the potential value of a visualization, i.e., its ability to provide a proper understanding of the data.
In S23, Lan et. al. conducted crowdsourcing studies to identify affective responses to infographics and derived practical design heuristics. They divided them into two categories: Usability and Expressiveness. The Usability heuristic comprises aspects like accessibility, readability, messaging, and credibility. The Expressiveness category includes embodiment, narrative, and uniqueness.
S14 introduces the concept of “purposeful visualizations” to highlight the importance of context and suggests a taxonomy to support the design and evaluation of such visualizations. The model covers seven assessments areas: visual representation, information presentation, psychology of the observer, information quality, visual impact, overall design style, and overall performance. Each area further breaks down into several subcategories.
The authors in S58 propose a method that consists of a usability questionnaire and a fuzzy logic application to evaluate participant's responses. The evaluation criteria are communication, which comprises satisfaction, flexibility, and learning; information, which includes accessibility, accuracy and searchability; and technology, which consists of learnability, efficiency, and error-proneness. Bai et. al. [S86] suggest measuring the Visual Intelligence Density (VID) of a visualization to quantify the amount of useful decision support information it provides and to evaluate its capabilities in terms of supporting decision making. The evaluation criteria used to measure VID is based on Tufte’s work in addition to their own proposed criteria: comparisons; causality, mechanism, structure, and explanation; multivariate analysis; integration of evidence; documentation; content counts most of all; 1D, 2D, 3D; variety of views; user interaction; animation. Each criterion is quantified in a scale from 0 (very weak capabilities) to 10 (extremely strong capabilities).
To measure visualization literacy, (i.e., the ability and skill to read and interpret visually represented data and extract information) Boy et. al. [S18] developed a method based on Item Response Theory for line charts, bar charts and scatter plots. The tasks considered to model test items were finding maximum, minimum, variations, intersections, calculating averages, and making comparisons. Each test item involved a stimulus, a task, and a question. On the same line, Lee et. al. [S59] developed the Visualization Literacy Assessment Test (VLAT), specially aimed at non-experts users. It consists of 12 data visualization types, namely: line chart, bar chart, stacked bar chart, 100% stacked bar chart, pie chart, histogram, scatter plot, area chart, stacked area chart, bubble chart, choropleth map, and treemap. The final test contains 53 multiple choice items that cover eight major tasks related to visualization. Each item is composed of a task, a content validity ratio (CVR) an item difficulty index (P) and an item discrimination index (D).
S61 presents VisEngage, a self-assessment questionnaire that comprises eleven characteristics of user engagement: aesthetics, captivation, challenge, control, discovery, exploration, creativity, attention, interest, novelty, and autotelism. The questionnaire consists of 22 questions rated on a seven-point Likert scale and it is intended to be used immediately after interacting with a visualization.
This section discusses the implications derived from our work. We reflect on the main findings of the research questions and address the threats to the validity of the results.
We found that even though the concepts of “data visualization” and “storytelling” are tightly associated when it comes to creating effective visuals, they are not interchangeable, and visualizing information does not necessarily mean creating a narrative visualization. To gain a thorough understanding of what data storytelling is, we articulate the difference between information visualization and storytelling below and derive a working definition of data storytelling.
Information Visualization
The term data visualization is used to describe the visualization of large data sets. Data visualizations were primarily meant for use in the scientific field and aimed at an efficient reading of data in analytical tasks [98]. Information visualization is a subfield of data visualization [1] that leverages the cognitive capacity of human visual perception, evolved for fast pattern detection and recognition, to communicate underlying relationships and trends in large datasets. The major goal of Information visualization is to amplify cognition [99], and help people perform tasks effectively [3].
The term “chart” or “visualization” describes a graphical representation of data. Charts, particularly within the context of presentation or persuasion, are designed to aid in the memorability of the presented data [100] and reduce cognitive load [99], [101], [102].
Storytelling and Narrative
Storytelling is a central aspect of human communication and cognition: for a long time, people have used stories to convey information, values, and experiences. In the field of research, there are many fragmented views and inconsistent definitions: some authors draw a distinction between “storytelling” and “narrative,” while others used the terms interchangeably.
Storytelling is defined as the social and cultural activity of sharing stories [103]. A fundamental aspect of storytelling are emotions and the cognitive responses the story evokes in its audience [104]. Gabriel [105] defined stories as “emotionally and symbolically charged narratives” oral or written that “usually have a plot, characters, aim to entertain, persuade or win over.”
According to the Oxford English Dictionary narrative is described as “an account of a series of events, facts, etc., given in order and with the establishing of connections between them.” It combines the narrative contents (story) and the narrative form (discourse) [104]. In simple terms, narration is the telling of a sequence of events to convey a story to an audience.
A well-told story conveys great quantities of information in relatively few words in a format that is easily assimilated [106].
Data Storytelling
Kosara and McKinlay [23] define a data story as an ordered sequence of steps consisting of visualizations that can include text and images but are essentially based on data.
Riche et. al. [22] refer to data-driven stories as stories that start from a narrative that is either based on or contains data, often portrayed by data visualizations, to clarify, inform and provide context to visually salient differences. In [20], the authors describe it as a sequence of “story pieces”, (facts backed up by data), visualized to support one or more intended messages. The visualization can include annotations (labels, pointers, text) or narration to highlight and emphasize the message and to avoid ambiguity. These story pieces are presented in a meaningful order to support the author’s main goal (educate, persuade, convince, etc.).
Based on these results we derive a definition of data storytelling, refining and incorporating previous ones, as:
The creation of narrative visualizations to convey an intended message, which can include images, text, and annotations to emphasize the message, avoid ambiguity, and facilitate decision making.
Fig. 6: Data Storytelling consists of three major elements: data, narrative, and visualization [107].
As [107] points out, data storytelling sits at the intersection between data, traditional narrative, and visualization (Fig. 6). This is also consistent with the work of Edmond and Bednarz [116], as they propose “NarVis” (narrative visualizations) are situated between data visualization and narrative. According to [108], the essence of a narrative visualization is good storytelling. A story worth telling challenges the reader and is a means of discovery. It drives the audience to ask more questions and pushes them from simply believing, to knowing with a degree of confidence [26].
The aim of this question was to collect and summarize existing guidelines for creating narrative visualizations, and when possible, to also explain how they are implemented. In general, we observed there is not a lack of guidelines, but rather they are scattered across the literature and some of them only apply to a certain type of chart.
The most frequently mentioned best practice was BP17, “choose the visualization technique that better supports the expected tasks.” It is also one of the practices for which we found the most implementations (over thirty), as it directly influences the amount of time it takes the user (or decision maker) to solve a problem and therefore, its complexity [118]. As discussed in several primary studies, different charts (or design choices within a single chart) perform better than others depending on the task, and designers must consider how they want the display to support a specific task, at potential cost for others [119]. For instance, spotting outliers in a scatterplot would be difficult at low marker opacity but estimating data density could benefit from it [59].
We believe this guideline is intrinsically related to BP5, (“select the appropriate visualization considering the types of data to represent and the advantages and disadvantages of each technique”), since one cannot choose the appropriate visualization without also considering the target tasks, both critical to aid the user in understanding the underlying data and improve decision making. BP5 however, also highlights the importance of tailoring visualizations to their audiences, considering aspects like chart familiarity and learning curves.
The second most referenced practice was BP8 “map information and data dimensions to the most salient visual features”, followed by BP15 “use text, labels and annotations for effective information consumption and decision making”. BP8 reflects on whether the visualization uses comprehensible data encodings, as suggested by [31]. Features include color, size, orientation, and shape. These allow the user to perform the required tasks effectively. BP15 focuses on enhancing the interpretability of the information depicted in the charts while also developing the narrative aspect. Titles and text are key to increase memorability. As pointed out in [41], a good title can make the difference between a visualization that is recalled correctly from one that is not. Labels, in turn, help orient the user and annotations can be used to highlight interesting patterns.
We found that many practices resemble user interface design guidelines, such as BP3, BP22, BP33, BP36 or BP37. Others, mainly focus on storytelling issues, such as BP25 and BP30. Moreover, we did not find any practical demonstrations for BP4, and BP26 and BP38 as they are straightforward guidelines and do not require illustration.
Overall, these findings implicate that each best practice may be associated with one or more evaluation criteria, given that each one serves a purpose (improving usability, increasing memorability, enhancing comprehension, etc.). Further analysis is necessary to validate these associations.
The goal of this question was to investigate the factors involved in quality visualizations, particularly, for narrative and storytelling purposes. This has been a topic of growing interest in the research community over the years. In general, a visualization is considered effective if it helps people extract accurate information [107] without further complexity [44].
Although several studies propose different sets of criteria, we found there is no unified standards for what constitutes an “effective” visualization, instead every author focuses on evaluating a given aspect of a visualization and the traits it encompasses. We collected the most mentioned, well-known criteria and grouped the less common features within these, as sub criteria.
Among the primary criteria we found Memorability, Comprehension and Engagement. Many studies emphasize the importance of memorability in visualizations, however, not every author agrees on it [72], arguing that the fact that the audience remembers a visualization, does not necessarily mean it is effective. Moreover, it represents certain challenges to measure it [48]. We believe it is a fundamental aspect, as it is essential to remember information prior to making decisions; particularly when the user has limited time to interact with the visualization (e.g., company meetings, crisis settings [120]).
Comprehension was pointed out by three studies as the primary goal of any visualization. It is the criterion for which we found most related items (16). Even though it is highly related to literacy, we did not include it as sub criteria since it is inherent to the user [121], rather than a visualization trait. Designers can tailor visualizations to support the audience’s various levels of literacy, thus making information as comprehensible as possible.
Engagement was pointed out by three studies as a complex construct involving several factors such as aesthetics, user control, or exploration, for instance. Even though it lacks a clear definition, its main concern is the user’s immersion in a visualization[122]. A visualization that is viewed for a certain amount of time and interacted with, will be considered as more successful. As Mahyar et. al. suggest, it is even more important when the target audience are not domain experts [123]. While there have been several efforts towards this direction, we found there is no unified approach to measure engagement.
Among the most frequently mentioned sub criteria was the “aesthetics” or style of visualizations. This makes sense, given it is tied to every major criterion: if the user finds the visualization aesthetically pleasing, the more willing it is to use it [124] (perceived usability), spend time on it (engagement) and remember the information (memorability).
Our findings differ slightly from what [73] proposes as criteria to evaluate data driven stories, as we did not find “dissemination” or “impact” per se in the primary studies. It can be argued, however, that these are highly related to engagement, and more studies are necessary to have a deeper understanding of it. Regarding impact, some of the studies mentioned in previous sections have tested the effect of incorporating storytelling in regular visualizations to measure the audience’s reaction[125], for instance, or the decision-making capabilities [14], [15].
Overall, as [73] points out, all these criteria are subjective constructs and thus, they depend on the context of application and cannot be measured directly. It is essential to consider the goal of a visualization when evaluating them.
The motivation behind this RQ was to find evaluation methodologies that are consistent with the best practices and criteria found in previous questions, outside of the traditional laboratory studies.
A variety of approaches have been proposed in that way. Some derive from the field of HCI, such as heuristics evaluation (S36, S70, S71, S23). Other models point to a specific criterion, such as comprehension (S31) or engagement (S61, S73) and others involve the use of algorithms (S58). There was only one method whose goal was to compare two different techniques and select the most appropriate (S67).
Among the heuristic evaluation, there were a few authors that suggest the standard technique be supplemented with new details, such is the case of the value-driven heuristics (S73) to assess the potential utility of a visualization, or S23, who focus on the “affective” impact of visualizations. Heuristic evaluation, however, has certain limitations, as it depends on the background and domain knowledge of the evaluators and cannot always be applied due to their generality.
We found that many approaches do not explicitly mention the targeted techniques, nor the goal of the visualizations they assess (decision making support, persuasion, etc.). Although we found several best practices in RQ2, we observed the evaluation strategies do not consider all of them, rather they focus only on a certain guideline or set of guidelines. For instance, the complexity score method in S28 primarily evaluates the aspect ratio of a chart, so that the user can perform tasks more efficiently, while S31 takes into account perceptual, cognitive and presentation aspects to assess comprehension.
We believe these methods may be classified into those that assess the visualization itself (S14, S23, S28, S31, S67, S36, S70, S71, S58, S86) and those that evaluate aspects concerning the user, such as literacy (S18, S59) or engagement (S61, S73).
In general, and in line with past research findings [126], the major obstacle to developers and designers of visualizations is the lack of out-of-the-box, ready to use evaluation tools. These methods, however, can serve as a starting point to other evaluation models.
Construct Validity: Construct validity is determined by our ability to capture what we intended. During the search, primary studies could have been missed. We mitigated this threat by searching on different libraries that cover the majority of the high-quality publications in SE and complemented the search with forward and backward snowballing sampling [37]. In addition, we performed an updated search re executing our search query to capture new papers published during the course of this research.
Internal Validity: These threats reflect on possible wrong conclusions when causal relations are examined [127]. Researcher bias constitutes a threat to the internal validity. To reduce this threat, we performed the selection process iteratively. For the data extraction phase, we conducted a pilot extraction to validate the data extraction form. We had one researcher extracting the data and another reviewing the extraction. Any conflicts during this phase were discussed and resolved by the authors. To measure the level of agreement between researchers, we used the Cohen Kappa statistic [83].
External Validity: External validity refers to what extent it is possible to generalize the findings. To ensure the widest coverage possible, we included papers published from 1984 to 2021. The excluded papers may affect the generalizability of our results. However, we argue that they do not have a significant impact on our review, as the ones included share similar ideas and recommendations.
Conclusion Validity: Conclusion validity measures the reproducibility of the study. This threat was mitigated by following the protocol proposed by [128], widely used in SE research, to determine research questions, data sources and search strategy, inclusion and exclusion criteria, quality assessment, data extraction and study selection.
This paper presented a systematic review of 95 studies about information visualization and data storytelling of the past 30+ years. We were interested in collecting existing definitions of “data storytelling” reported in the literature, guidelines for the design of narrative visualizations as well as evaluation criteria and the methods to assess them.
Our findings showed there is no clear, agreed-upon definition of what “data storytelling” encompasses, though several efforts are being made in that direction. We contribute by deriving a working definition of it, distinguishing among the concepts involved.
Furthermore, the results of this SLR provide a useful overview of design guidelines for narrative visualizations that can serve as a starting point to assist practitioners and researchers to create effective communications.
Regarding evaluation criteria and methods, we noticed that although there are useful approaches to assess visualizations, some problems still remain since not all of them are comprehensive enough in terms of the criteria and best practices they consider and thus do not support the real needs of designers and developers.
One limitation of past research is that they not always stated the ultimate goal of the visualizations they assessed, (a critical aspect to interpret results accordingly), or they only focused on a certain type of chart. Many studies observed that evaluating visualizations is challenging because it involves subjective constructs. Future research could focus on characterizing and defining the criteria identified in this work to gain a better understanding on how to measure them.
Based on the results of this SLR, we plan to develop an evaluation model for narrative visualizations that captures the guidelines and criteria discussed as comprehensively as possible to enable an iterative assessment of visualizations during the design phase.
Table A1: List of selected primary studies.
| ID | Ref | Paper Title |
|---|---|---|
| S01 | [129] | Graphical Encoding for Information Visualization: An Empirical Study |
| S02 | [19] | Narrative Visualization: Telling Stories with Data |
| S03 | [21] | Visualization Rhetoric: Framing Effects in Narrative Visualization |
| S04 | [130] | Suggested Interactivity: Seeking Perceived Affordances for Information Visualization |
| S05 | [131] | Graph and chart aesthetics for experts and laymen in design: The role of familiarity and perceived ease of use |
| S06 | [132] | A Study on Designing Effective Introductory Materials for Information Visualization |
| S07 | [14] | The Attraction Effect in Information Visualization |
| S08 | [133] | Extending Open Data Platforms with Storytelling Features |
| S09 | [134] | Evaluating Visualizations Based on the Performed Task |
| S10 | [57] | Improving 2D scatterplots effectiveness through sampling, displacement, and user perception |
| S11 | [135] | Investigating the Comprehension Support for Effective Visualization Tools – A Case Study |
| S12 | [61] | Perceptual Guidelines for Creating Rectangular Treemaps |
| S13 | [33] | Whisper, Don’t Scream: Grids and Transparency |
| S14 | [91] | Purposeful Visualization |
| S15 | [106] | A Study on Dual-Scale Data Charts |
| S16 | [46] | An Empirical Study on Using Visual Embellishments in Visualization |
| S17 | [58] | Selecting the Aspect Ratio of a Scatter Plot Based on Its Delaunay Triangulation |
| S18 | [136] | A Principled Way of Assessing Visualization Literacy |
| S19 | [40] | Clutter-Aware Label Layout |
| S20 | [59] | Towards Perceptual Optimization of the Visual Design of Scatterplots |
| S21 | [137] | Evaluating Cartogram Effectiveness |
| S22 | [39] | Rainbows Revisited: Modeling Effective Colormap Design for Graphical Inference |
| S23 | [92] | Smile or Scowl? Looking at Infographic Design Through the Affective Lens |
| S24 | [138] | Visual Reasoning Strategies for Effect Size Judgments and Decisions |
| S25 | [62] | Improving the Visualization of Hierarchies with Treemaps: Design Issues and Experimentation |
| S26 | [139] | Evaluating Visual Table Data Understanding |
| S27 | [140] | Effects of 2D Geometric Transformations on Visual Memory |
| S28 | [141] | Complexity Analysis for Information Visualization Design and Evaluation |
| S29 | [142] | The Effect of Aesthetic on the Usability of Data Visualization |
| S30 | [143] | Perceptual Dependencies in Information Visualization Assessed by Complex Visual Search |
| S31 | [93] | Comprehension of Visualization Systems - Towards Quantitative Assessment |
| S32 | [60] | Evaluation of Symbol Contrast in Scatterplots |
| S33 | [144] | Evaluating the Effectiveness and Efficiency of Visual Variables for Geographic Information Visualization |
| S34 | [47] | Useful Junk? The Effects of Visual Embellishment on Comprehension and Memorability of Charts |
| S35 | [118] | Using Cognitive Fit Theory to Evaluate the Effectiveness of Information Visualizations: An Example Using Quality Assurance Data |
| S36 | [145] | An Heuristic Set for Evaluation in Information Visualization |
| S37 | [146] | Graphical Perception of Multiple Time Series |
| S38 | [147] | Eye tracking for visualization evaluation: Reading values on linear versus radial graphs |
| S39 | [34] | The Effect of Colour and Transparency on the Perception of Overlaid Grids |
| S40 | [148] | Comparing Averages in Time Series Data |
| S41 | [149] | Towards Adaptive Information Visualization: On the Influence of User Characteristics |
| S42 | [44] | How Capacity Limits of Attention Influence Information Visualization Effectiveness |
| S43 | [150] | Evaluating the Effect of Style in Information Visualization |
| S44 | [151] | Individual User Characteristics and Information Visualization: Connecting the Dots through Eye Tracking |
| S45 | [152] | Evaluation of Alternative Glyph Designs for Time Series Data in a Small Multiple Setting |
| S46 | [153] | Data Visualisation, User Experience and Context: A Case Study from Fantasy Sport |
| S47 | [88] | A Deeper Understanding of Sequence in Narrative Visualization |
| S48 | [48] | What Makes a Visualization Memorable? |
| S49 | [154] | Sample-Oriented Task-Driven Visualizations: Allowing Users to Make Better, More Confident Decisions |
| S50 | [52] | Improving Information Perception of Graphical Displays – an Experimental Study on the Display of Column Graphs |
| S51 | [119] | Task-Driven Evaluation of Aggregation in Time Series Visualization |
| S52 | [155] | Would you prefer pie or cupcakes? Preferences for data visualization designs of professionals and laypeople in graphic design |
| S53 | [156] | Evaluation of information visualization techniques: analysing user experience with reaction cards |
| S54 | [157] | Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error |
| S55 | [45] | The relation between visualization size, grouping, and user performance |
| S56 | [49] | An Evaluation of the Impact of Visual Embellishments in Bar Charts |
| S57 | [41] | Beyond Memorability: Visualization Recognition and Recall |
| S58 | [97] | Towards an intelligent evaluation method of medical data visualizations |
| S59 | [121] | VLAT: Development of a Visualization Literacy Assessment Test |
| S60 | [15] | Narratives in Crowdsourced Evaluation of Visualizations: A Double-Edged Sword? |
| S61 | [96] | Assessing User Engagement in Information Visualization |
| S62 | [158] | Visual Narrative Flow: Exploring Factors Shaping Data Visualization Story Reading Experiences |
| S63 | [159] | Aesthetic Experimental Study on Information Visualization Design Under the Background of Big Data |
| S64 | [36] | Modeling Color Difference for Visualization Design |
| S65 | [103] | Conceptual and Methodological Issues in Evaluating Multidimensional Visualizations for Decision Support |
| S66 | [160] | Assessing the Graphical Perception of Time and Speed on 2D+Time Trajectories |
| S67 | [161] | Line Graph or Scatter Plot? Automatic Selection of Methods for Visualizing Trends in Time Series |
| S68 | [162] | Evaluating Interactive Graphical Encodings for Data Visualization |
| S69 | [53] | What’s the Difference? Evaluating Variants of Multi-Series Bar Charts for Visual Comparison Tasks |
| S70 | [163] | The Development of Heuristics for Evaluation of Dashboard Visualizations |
| S71 | [164] | An Instrument for Evaluating the Quality of Data Visualizations |
| S72 | [54] | Improving Perception Accuracy in Bar Charts with Internal Contrast and Framing Enhancements |
| S73 | [102] | A Heuristic Approach to Value-Driven Evaluation of Visualizations |
| S74 | [37] | Mapping Color to Meaning in Colormap Data Visualizations |
| S75 | [165] | Face to Face: Evaluating Visual Comparison |
| S76 | [166] | Understanding Visual Cues in Visualizations Accompanied by Audio Narrations |
| S77 | [167] | Comparing the effectiveness of visualizations of different data distributions |
| S78 | [50] | A Baseline Study of Emphasis Effects in Information Visualization |
| S79 | [38] | The Effect of Color Scales on Climate Scientists’ Objective and Subjective Performance in Spatial Data Analysis Tasks |
| S80 | [125] | Structure and Empathy in Visual Data Storytelling: Evaluating their Influence on Attitude |
| S81 | [43] | Declutter and Focus: Empirically Evaluating Design Guidelines for Effective Data Communication |
| S82 | [168] | Graph Design: The Data-ink Ratio and Expert Users |
| S83 | [42] | Minimalism and the Syntax of Graphs |
| S84 | [31] | Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods |
| S85 | [169] | Exploring the Role of Individual Differences in Information Visualization |
| S86 | [170] | Visual Intelligence Density: Definition, Measurement, and Implementation |
| S87 | [171] | Task-Based Effectiveness of Basic Visualizations |
| S88 | [172] | The use or misuse of three-dimensional graphs to represent lower-dimensional data |
| S89 | [55] | Four Experiments on the Perception of Bar Charts |
| S90 | [35] | How Colors in Business Dashboards Affect Users’ Decision Making |
| S91 | [173] | Improving Visualization Design for Effective Multi-Objective Decision Making |
| S92 | [174] | The Unmet Data Visualization Needs of Decision Makers within Organizations |
| S93 | [175] | Remote Usability Assessment of Topic Visualization Interfaces with Public Participation Data: A Case Study |
| S94 | [56] | Visual Arrangements of Bar Charts Influence Comparisons in Viewer Takeaways |
| S95 | [51] | Which Emphasis Technique to Use? Perception of emphasis techniques with varying distractors, backgrounds and visualization types |
Table A2: Quality assessment results.
| ID | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 | Q11 | Quality |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| S01 | 1,0 | 1,0 | 0,5 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 0,0 | 0,0 | 1,0 | 73% |
| S02 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | N/A | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 95% |
| S03 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | N/A | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 95% |
| S04 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S05 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 95% |
| S06 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 0,5 | 1,0 | 86% |
| S07 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S08 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | N/A | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 95% |
| S09 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 0,5 | 0,5 | 0,5 | 0,0 | 0,0 | 1,0 | 64% |
| S10 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 0,0 | 1,0 | 1,0 | 0,5 | 1,0 | 77% |
| S11 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 0,5 | 1,0 | 82% |
| S12 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 1,0 | 95% |
| S13 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 0,5 | 1,0 | 91% |
| S14 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | N/A | 0,0 | 1,0 | 1,0 | 0,0 | 1,0 | 75% |
| S15 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 95% |
| S16 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 1,0 | 0,0 | 1,0 | 82% |
| S17 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 0,0 | 1,0 | 73% |
| S18 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S19 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 0,0 | 0,5 | 1,0 | 73% |
| S20 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S21 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S22 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S23 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S24 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 1,0 | 95% |
| S25 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 0,0 | 0,0 | 1,0 | 77% |
| S26 | 1,0 | 0,0 | 1,0 | 1,0 | 1,0 | 0,5 | 0,0 | 1,0 | 0,0 | 0,0 | 1,0 | 59% |
| S27 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 86% |
| S28 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 1,0 | 0,0 | 0,0 | 1,0 | 68% |
| S29 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 0,5 | 1,0 | 1,0 | 0,0 | 1,0 | 82% |
| S30 | 1,0 | 0,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 82% |
| S31 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 0,0 | 1,0 | 82% |
| S32 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 0,0 | 1,0 | 82% |
| S33 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 95% |
| S34 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 95% |
| S35 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S36 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 86% |
| S37 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S38 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S39 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S40 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S41 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S42 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 0,0 | 1,0 | 82% |
| S43 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 0,5 | 0,0 | 1,0 | 82% |
| S44 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S45 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S46 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 0,5 | 1,0 | 0,5 | 0,0 | 1,0 | 77% |
| S47 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S48 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S49 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S50 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S51 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S52 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S53 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 0,5 | 0,0 | 1,0 | 1,0 | 0,0 | 1,0 | 73% |
| S54 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S55 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S56 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S57 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 95% |
| S58 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 0,0 | 1,0 | 82% |
| S59 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S60 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S61 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S62 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S63 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 0,0 | 0,0 | 1,0 | 73% |
| S64 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S65 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S66 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S67 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 1,0 | 95% |
| S68 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S69 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S70 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 95% |
| S71 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 0,0 | 1,0 | 82% |
| S72 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 0,0 | 1,0 | 82% |
| S73 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S74 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S75 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S76 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 91% |
| S77 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S78 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S79 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S80 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S81 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S82 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | N/A | 1,0 | 1,0 | 1 | 0 | 1,0 | 90% |
| S83 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S84 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 0,5 | 1,0 | 91% |
| S85 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 0,0 | 1,0 | 82% |
| S86 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | N/A | 1,0 | 1,0 | 0,5 | 0 | 1 | 85% |
| S87 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S88 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S89 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 95% |
| S90 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 0,0 | 0,0 | 1,0 | 77% |
| S91 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,5 | 1,0 | 95% |
| S92 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 1,0 | 0,5 | 0,5 | 1,0 | 82% |
| S93 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 0,0 | 1,0 | 91% |
| S94 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 100% |
| S95 | 1,0 | 1,0 | 0,5 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 1,0 | 95% |